Constraining of weights using regularities

In this paper we study how global optimization methods (like genetic algorithms) can be used to train neural networks. We introduce the notion of regularity, for studying properties of the error function that expand the search space in an artiicial way. Regularities are used to generate constraints on the weights of the network. In order to nd a satissable set of constraints we use a constraint logic programming system. Then the training of the network becomes a constrained optimization problem. We also relate the notion of regularity to so-called network transformations. 1. Introduction The training of a neural network consists of nding a set of weights that minimizes the neural network's error criterion. Most of the standard methods use local gradient search techniques. Often this works well, but there are cases in which it is problematic, for example for recurrent networks, or networks with non-diierentiable error criteria. Nevertheless, this type of networks can be very useful in applications (for example in applications based on time sequences). Moreover, a local method can get stuck in local minima. Therefore, alternative approaches based on global optimization methods have been proposed. However , there is a well-known problem with the application of global optimization methods to the training of neural networks. Firstly, these methods can be relatively slow. It depends on the application whether this is a problem or not. Secondly, there is the so-called \competing conventions problem" (see e.g. 3,4]). When one chooses a representation (in this case for a neural network), then it can be the case that the same individual has more than one representation. This enlarges (in an artiicial way) the search space. The standard approach is to nd a clever representation such that this does not happen. In this paper we take a diierent approach. We see the problem as a constrained optimization problem, and through the constraints we avoid the \competing conventions problem". First, the notion of regularity is introduced for studying properties of the error function of the networks. Next, these properties are used